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(57) Abstract 



A system for implementing controls on a 
software interface. Each control is formed by 
an image layer which is visible on the interface, 
and one or more masks which are linked to 
control instructions. The masks are generally 
bitmaps in which colour information determines 
the instructions. A user positions an interface 
device such as a mouse pointer over appropriate 
parts of the image to request an action. The 
position of the pointer is related to each mask 
and an instruction is determined according to the 
colour of the mask at that position. 
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IMPROVEMENTS RELATING TO CONTROL OF COMPUTER INTERFACES 
FIELD OF THE INVENTION 

5 This invention relates to user control of graphical computer interfaces and 
particularly but not solely to control systems involving images which are displayed 
for a user on the screen of a computer system. The invention also relates to retrieval 
and presentation of information on such an interface. 

0 BACKGROUND TO THE INVENTION 

Graphical interfaces are included with most computer operating systems such as the 
WindoWSTm and Macintosh' systems. Controls are generally provided on each 
interface by way of active screen areas which contain more or less complex imaaes. 
5 commonly referred to as buttons. Users generally actuate the controls using a screen 
pointer and a manual pointer controller such as a mouse. Positioning the pointer on 
a control image and clicking the mouse causes an event such as display of additional 
information on the interface. 

1 Controls are typically defined as a square or rectangular area on the screen havins 
an origin point, a width and a depth. Each area typically represents a single control 
and a single event which may be generated by the user. A meaningful imaae is 
usually associated with each control, such as the shape of a button.' and is often 
displayed with a realistic three dimensional appearance. Pixel data representing each 
image is stored as a graphical file which usually includes a bitmap. Several versions 
of each image are usually stored and displayed alternately as a particular control is 
selected and operated by the user, to create visual effects such as a button being 
highlighted, pushed down, or raised from the screen for example. 

Various routines run within the graphical interface software to monitor the position 
of the pointer on the screen, which may take the form of an arrow for example. Each 
movement of the pointer generates an interrupt and returns the position of a 
reference point on the pointer, such as the screen coordinates of the tip of the arrow. 
If the reference point is positioned in a control area then the control image associated 
with that area may be varied in some way. Each actuation of the pointer controller 
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generates an interrupt causing assessment of the current position of the pointer and 
determination whether an event must take place. If the pointer is positioned on a 
control area then the image associated with that area is further varied and the event 
associated with that area is generated. 

Applications which may be run by the user under the operating system often require 
complex controls. Numerous alternative control events must often be presented side 
by side for selection by the user within a relatively small screen area. The 
possibilities for doing so are complicated or limited by the traditional use of a single 
square or rectangular area to represent a single control. The possibilities are also 
limited by the practice of displaying a relatively sophisticated image in relation' to 
each control, as this prevents pixel sensitive control information being derived from 
the position of the pointer on the image. 

Another limitation on traditional interface control systems arises where multiple 
events are to be triggered by operation of a single control. Similarly, different 
combinations of related events may be triggered by operation of closely spaced 
controls. These effects require special sensitivity in the arrangement of controls on 
the interface. A particular application has arisen in relation to display of various 
combinations of database information for computer systems operated by utility 
organisations such as electricity supply companies. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to provide for interface control systems which 
are able to conveniently incorporate multiple and perhaps irregularly shaped controls 
within a given interface area, or at least to provide an alternative to existing systems. 
In general terms the invention generally involves use of one or more control masks 
in association with a control image, and use of pixel based control information 
derived from the masks to determine the events which are generated on activation 
of the particular control. 

Accordingly in one aspect the invention may broadly be said to consist in a control 
system for a graphical user interface, including: a control image having one or more 
image portions which are displayed on the interface, a control mask having one or 
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more mask portions which correspond at least in part to respective image portions, 
a first process which determines control information from the mask when an' 
interface pointer is positioned on the control image by an operator, and a second 
process which carries out one or more first predetermined actions on the interface 
5 according to the control information. 

In another aspect the invention may be said to consist in a method of operating a 
computer control on a graphical user interface, including: displaying a control image 
on the interface, determining position information for an interface pointer which is 
10 actuated by an operator, detecting when the pointer is positioned on the control 
image, determining control information from a control mask which corresponds at 
least in part to the image, and varying the control image according to the control 
information. 



15 



20 



In a third aspect the invention may be said to consist in a method of presenting 
information using a control system on a graphical user interface, including: 
determining position information for a location on the interface, determining control 
information from a plurality of control masks which include the position on the 
interface, selecting information items from a plurality of sources corresponding to 
respective control masks in the control system, and displaying the information items 
on the interface or carrying out some other action. 



The invention may also broadly be said to consist in any alternative combination of 
pans or features here referred to or shown in the accompanying drawings. Known 
25 equivalents of these parts or features not expressly set out are nevertheless deemed 
to be included. 



BRIEF DESCRIPTION OF THE DRAWINGS 



30 Preferred embodiments of the invention will be described with respect to the 
drawings of which: 

Figure 1 shows a control image having two portions displayed on a graphical 
user interface, by way of example. 

Figure 2 shows a control mask which corresponds to the displayed image in 
35 accord with the invention. 
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Figures 3a and 3b show an interface pointer positioned over alternative 
portions of the control image which are correspondingly highlighted. 

Figures 4a and 4b show alternative portions of the image displayed in up and 
down forms when selected by actuation of a pointer controller. 

Figure 5 outlines a control process which causes highlighting of the image 
portions on movement of the pointer. 

Figure 6 outlines a control process which causes image portions to be 
displayed in the down form on actuation of the pointer controller. 

Figure 7 outlines a process which causes an event on the interface according 
to control information in the mask. 

Figure 8 outlines a process which enables events to take place according to 
control information in a plurality of masks which correspond to the control image. 

Figures 9a to 9f show a simplified control image and a plurality of mask 
images relating to the control image, for a geographical application by way of 
example. 

Figure 10 is a display which might be generated by actuation of a portion of 
the control image in Figure 9a at a particular geographical location. 

Figure 1 1 is a personal computer system on which a graphical user interface 
may be implemented, also by way of example, and 

Figure 12 schematically shows a computer system including an operating 
system, graphical interface and database arrangement. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Referring to these drawings it will be appreciated that the invention may be 
implemented in various forms and as pan of a wide range of graphical interface 
applications. The following description is given by way of example only. Operation 
of those parts of a graphical interface which are standard in nature will be well 
understood by a skilled person and need not be described in detail. 

Figure 1 shows a reasonably complex control image 10 which could be associated 
with a control system in a graphical interface, according to the invention. The image 
is a multicoloured double spiral formed by two interleaved arms 1 1 and 12 on a plain 
background, and is designed to have a three dimensional aspect which appears raised 
from the interface. Graphical information which determines the image is stored as 
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a square bitmap indicated in dashed outline. The information includes data which 
determines the colour of each pixel within the image when displayed in an area on 
a computer screen as part of a control. Due to the complex visual nature of the image 
each portion is represented by a range of colours as required to create the various 
visual effects. Each arm of the image is a separate image portion which represents 
a separate control area having a spiral shape and is separated from the other arm by 
a non-control area 13. Separate controls which can be operated to generate distinct 
events can be thereby provided for the user. A two part control of this kind would 
be difficult to implement using conventional control systems. 

Figure 2 is a simplified mask image 20 which may be associated with the control 
image 10 of Figure 1. The mask includes separate mask portions 21 and 22 which 
correspond in shape and size to arms 1 1 and 12 of the image 10. A plain portion 23 
corresponds to portion 13 which separates the arms. The mask is not displayed to a 
user and each of the mask portions is represented by a single colour. Graphical 
information which determines the mask image is stored as a bitmap indicated in 
dashed outline. The single colour which is allocated to each portion of the mask 
forms control information for the control system which incorporates the image 10. 
The colour may be considered a code which is used by a control process to generate 
the event which is associated with the control. It should be noted that the^shapes. 
sizes, colours and locations of the mask portions of image 20 preferably correspond 
closely with the control portions of image 10 but need not necessarily do so. 

Figures 3a and 3b are control images 30 and 31 which might be displayed in 
response to the position of an interface pointer 35 in the form of an arrow. In Figure 
3 a the pointer has been moved by the user to a position on image portion 12 and the 
control system has responded by changing the colour of that portion to provide 
visual confirmation. In Figure 3b the pointer has been positioned on portion 1 1 and 
the control system has responded by changing the colour of that portion. It is the tip 
36 of the arrow which is used by the control system as a reference point when 
determining the position of the pointer at any instant. An action of this kind provides 
a highlight for the user and confirms that actuation of the pointer controller will 
generate the event which is associated with the particular control. Positioning the 
pointer on portion 13 produces no change in the image to indicate a non-active area. 
Various highlight actions are possible, including a conversion of the pointer by 
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colour or shape, as will be familiar to a skilled reader. The pointer could be 
converted to a cross for example. 

Figures 4a and 4b are control images 40 and 41 which might be displayed by the 
control system in response to actuation of the pointer controller in Figures 3a and 3b. 
In Figure 4a the controller has been actuated with the pointer 35 positioned on image 
portion 12, typically by depressing a mouse button, and that portion of the image 40 
is now displayed with an aspect which appears inwards on the screen. In Figure 4b 
the pointer controller has been actuated with the pointer positioned on portion 1 1 and 
that portion has been similarly depressed. Depression and release of a mouse button 
are sometimes termed mouse down and mouse up actions respectively. It is generally 
the mouse up action which generates an event associated with the particular control 
although a detail of this kind can vary among different interfaces. 

Figure 5 is a process flowchart which outlines how a control system may vary a 
control image to highlight placement of the pointer 35 on an image portion, such as 
described in relation to Figures 3a and 3b. Movement of the pointer controller is 
detected in step 50 and the system then determines whether the pointer is positioned 
on a control image or other control area of the interface in step 51. If the pointer is 
positioned on a control image the system retrieves control information from the 
associated mask in step 52. Control information is preferably pixel colour data from 
the corresponding position in a bitmap which forms the mask. If in step 53 the 
control information indicates that the pointer is positioned on an active image 
portion then in step 54 the system varies the control image, or perhaps the 
appearance of the pointer. 

Figure 6 is a process flowchart which indicates how a control system may vary a 
control image to indicate a mouse down event as described in relation to Figures 4a 
and 4b. Actuation of the pointer controller is detected in step 60 and the system then 
determines whether the pointer is positioned on a control image or other control area 
of the interface in step 61. If the pointer is positioned on a control image the system 
retrieves control information from the associated mask in step 62. If in step 63 the 
control information indicates that the pointer is positioned on an active image 
portion then in step 64 the system varies the control image to confirm that the mouse 
down event has taken place. This indicates to the user that on further actuation of the 
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controller, typically by releasing a mouse button as described in relation to Figure 
7, a control event will be generated. 

Figure 7 is a process flowchart which indicates how a control system may retrieve 
control information from a mask on detection of a mouse up event and trigger a 
control event as described in relation to Figure 10. The mouse event is detected in 
step 70 and the system determines whether the pointer is still positioned on a control 
image in step 7 1. If the pointer is still on a control image then control information 
is retrieved from the corresponding mask in step 72 and compared with previous 
control information in step 73. The pointer is normally required to be positioned on 
the same control for both the mouse down and mouse up events. If the pointer has 
moved elsewhere on the interface then any potential action arising from the mouse 
down event is preferably cancelled with an appropriate visual confirmation for the 
user. If the control information and therefore the pointer position is consistent the 
control system varies the control image in step 74. generally by raising a depressed 
portion, and then highlighting the portion. The image might be restored to its 
original appearance, for example. In step 75 the event associated with the control is 
generated, such as a display of information for the user. 

Figure 8 is a flowchart which outlines how the process of Figure 7 may be extended 
to trigger multiple control events as described in relation to Figure 10. More than 
one mask may be associated with a particular control on the interface. Each mask 
provides control information for a respective event according to the position of the 
pointer in the control image and the colour of the pixel at the corresponding position 
in the mask. Following step 75 in Figure 7, step 80 in Figure 8 determines whether 
any further masks are associated with the control image. If a further mask exists then 
control information is retrieved from the bitmap in the mask in step 8 I. The 
associated event is then generated in step 82. The process terminates once each 
associated mask has been considered. 



Figure 9a is a control image 90 which might be used in association with a 
geographical control system for an interface relating to New Zealand. The interface 
might be provided as part of a website containing New Zealand related information, 
for example, a kiosk at a public library, or a private computer system operated by a 
utility organisation. The image has been considerably simplified for purposes of 
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clarity and would show detailed topographical or municipal features in practice, for 
example. Pointer 35 is shown positioned by a user near a northern part of the South 
Island. The user may be seeking or providing data relating to an individual address 
or a physical site located in that part of the country. The data could be related to 
subscription services provided to that address by the utility organisation, for 
example. In the example given below, the data is geophysical and demographic in 
nature. A control image of this kind might cover most or all of an entire computer 
screen in area. 

Figures 9b to 9f are a series of mask images 91 to 95 associated with the control 
image 90. As bitmaps they may be considered as being superimposed on a common 
screen area. Each mask represents the distribution of a particular characteristic or 
statistic of New Zealand, once again by way of example. Each mask has a number 
of portions which in turn represent regions where the feature or statistic takes a 
common value, or a value within a predetermined range, with the values and perhaps 
other information being stored in a database. The mask in Figure 9E represents a 
distribution of soil types, while that in Figure 9F represents population statistics, and 
the others are also self evident. Regions having a common soil type or population 
density in a given range are represented in the mask by a common colour. The 
arrangement of coloured portions in these masks will generally not be indicated in 
the control image of Figure 9a. although images of this kind may well be accessible 
elsewhere. 

In Figure 9a the control image 90 has two active control portions 97 and 98 which 
represent the North and South Islands separated by a non-active portion. A user 
positioning a pointer or actuating the pointer controller on either portion will observe 
the effect of processes such as those described in relation to Figures 5 to 8. The 
image portion corresponding to the North Island may be highlighted, depressed and 
restored to its original appearance by movement and actuation of the pointer 
controller, for example. Each mask image has a number of distinct coloured portions 
which correspond to only part of the portions of image 90. However, in this example 
each mask portion is wholly within the area of an image portion, although this need 
not generally be the case. Alternatively, each control image portion may be wholly 
within the area of a respective mask portion. 
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Figure 10 is a display table 100 which might be generated on the interface by a user 
when actuating the pointer controller at a position as indicated in Figure 9a. The 
table contains individual elements of data retrieved from a database according to 
control information from each of the masks in Figures 9b to 9f. Each retrieval of a 
data element such as a level of rainfall is achieved as an event generated according 
to the processes of Figures 7 and 8. It will be appreciated that a very wide range of 
effects can be conveniently created on an interface or within a computer system 
using a control system of the present kind. Other actions may also take place in the 
computer system instead of or in addition to the display of information. Information 
could be added to the database for example, or a fresh screen display could be 
produced. 



Figures 1 1 and 12 indicate in highly schematic forms, hardware and software aspects 
of a computer system which might implement one or more interface control systems 
5 according to the invention. The hardware and software which is shown in these 
systems will be self evident to a skilled reader. It will also be appreciated that these 
systems are a few of many possible examples, only some of which have been 
mentioned above. 
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CLAIMS: 

1 . A control system for a graphical user interface, including: 

a control image having one or more image portions which are displayed on the 
interface, 

a control mask having one or more mask portions which correspond at least in 
part to respective image portions, 

a first process which determines control information from the mask when an 
interface pointer is positioned on the control image by an operator, and 

a second process which carries out one or more first predetermined actions on 
the interface according to the control information. 

2. A system according to claim 1 further including: 

a plurality of control masks each having one or more portions which 
correspond at least in pan to image portions, 

wherein the first process determines control information from each of the 
masks when the interface pointer is positioned on the control image. 

3. A system according to claim 2 further including: 

a third process which carries out one or more second predetermined actions 
according to the control information when a pointer controller is actuated by the 
operator. 

4. A system according to claim 3 further including: 

a plurality of information sources which correspond to the plurality of control 
masks. 

wherein the second predetermined actions include selection and display of 
information from the sources according to the control information. 

5. A system according to claim 1 wherein: 

the first predetermined actions include display of one or more alternative 
control images on the interface. 
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6. A system according to claim 1 wherein: 

the control mask includes a bitmap and the control information includes pixel 
colour data. 

7. A system according to claim 1 wherein: 

the control mask is a graphical image file derived from the control image. 

8. A method of operating a computer control on a graphical user interface, 
including: 

displaying a control image on the interface, 

determining position information for an interface pointer which is actuated by 
an operator. 

detecting when the pointer is positioned on the control image, 
determining control information from a control mask which corresponds in part 
to the image, and 

varying the control image according to the control information. 

10. A method according to claim 8 further including: 

detecting actuation of a pointer controller by the operator, and 

presenting information on the interface for the operator according to the control 

information. 

10. A method according to claim 8 further including: 

determining control information from a plurality of control masks which 
correspond to the control image. 

11. A method according to claim 10 further including: 

selecting and presenting information on the interface from a plurality of 
sources corresponding to the plurality of control masks. 

12. A method according to claim 8 further including: 

varying the control image by selecting an alternative image from a plurality of 
images. 
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13. A method according to claim 8 wherein: 

the position of the pointer on the interface is provided as pixel coordinate 
information. 

5 14. A method according to claim 8 wherein: 

the control mask is a bitmap and the control information includes pixel colour 

data. 

15. A method according to claim 1 1 wherein: 

10 the control image represents a geographical region and the plurality of sources 

contain information relating to operation of utilities within the region. 

16. A method of processing information using a control system on a graphical user 
interface, including: 

15 determining position information for a location on the interface, 

determining control information from a plurality of control masks which 
include the position on the interface, 

selecting information items from a plurality of sources corresponding to 
respective control masks in the control system, and 
20 earning out a computer action related to the information items. 

17. A method according to claim 16 further including: 

displaying a control image on the interface representing a geographical region, 
receiving geographical coordinate information relating to an activity at a site 
25 within the region, and 

converting the coordinate information into interface position information 
relating to the site as located on the control image. 

18. A method according to claim 16 further comprising: 
30 displaying the information items on the interface. 

19. A control system substantially as herein described with reference to the 
accompanying drawings. 

35 20. A method of operating a control system substantially as herein described with 
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reference to the accompanying drawings. 

21. A method of presenting information substantially as herein described with 
reference to the accompanying drawings. 

5 
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